Skip to content

PQ P5: hybrid key schedule (mlkem_ss || x25519_ss) — first X25519MLKEM768 handshake - #315

Merged
JC-000 merged 4 commits into
masterfrom
feat/pqc-p5-hybrid-keyschedule
Oct 18, 2026
Merged

JC-000 merged 4 commits into
masterfrom
feat/pqc-p5-hybrid-keyschedule

Conversation

@JC-000

@JC-000 JC-000 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Scope

PQ phase P5. Under PQ_HYBRID=1 the key schedule now follows the group the ServerHello chose: tls_kex_group == $EC extracts the handshake secret from the 64 B mlkem_ss || x25519_ss (draft-ietf-tls-ecdhe-mlkem order, ML-KEM first); $1D is unchanged. After #314 an 11EC server failed at the first encrypted record; now the handshake completes and the GET returns. Both shared secrets are zeroed as soon as the handshake secret exists.

Changes: src/tls_keyschedule.s (the IKM choice and the zeroing, all .ifdef PQ_HYBRID), src/crypto/pq_hybrid.s (outer php/plp removed from pq_sh_decaps), the oracle tools/pq_keylog_check.py, a new suite tools/test_pq_keyschedule.py, test_pq_kem.py [4] extended, 12 new mutants in tools/mutate_pq_sh.py, openssl_listener.py --groups default / --trace, two diag lines in the rig, CLAUDE.md (terse) and engineering-notes.

Decisions

  • Small concat, not a contiguous span. tls_mlkem_ss is in NET_BSS_TAIL and tls_shared_secret in data.s BSS, so the 64 B IKM is copied into input_buffer (dead scratch since the early-secret step) and the HKDF IKM pointer/length retargeted. hkdf_extract copies the IKM into hmac_data_buf (97 B) and HMACs it as the message (the salt is the key), so 64 is within its limit.
  • Zeroing at the extract, for both groups: tls_shared_secret, tls_mlkem_ss, input_buffer[0:64], hmac_data_buf[0:64]. Later HKDF calls reuse hmac_data_buf, so the test checks it by content (no 8-byte window of the IKM survives); mutant K11 shows bytes around offset 55-63 do survive without the zeroing. 'Q' spans unchanged and still correct (the seed+ss span just holds zeros earlier).
  • Other P4 residue, reported not changed: tls_ecdhe_privkey stays until 'Q' (same as classic builds); SHA-256's internal block/schedule buffers were not inspected (the next HMACs of the same routine rewrite them).
  • Oracle redesigned (the ss are gone): pq_keylog_check.py --state now compares tls_c_hs_secret/tls_s_hs_secret (reused for the application traffic secrets after tls_derive_traffic_keys) with the keylog's CLIENT_TRAFFIC_SECRET_0/SERVER_TRAFFIC_SECRET_0. Those are a pure function of the hybrid handshake secret and the transcript through the server Finished. Gate: tls_reached_connected == 7 (tls_state has no high-water latch, so the rig's CONNECTED oracle is outrun at turbo #204), tls_kex_group == $EC, tls_sh_err == 0, secrets non-zero; any non-zero seed/ss/shared-secret in a connected run is a RESIDUE failure (exit 1). The explicit-argument P4 mode is unchanged. PQ_DUMP now also dumps the two secrets and the latch.
  • pq_sh_decaps: redundant outer php/plp removed (pq_mlkem_scrub saves/restores P; pha/pla kept). Mutants Y3p/Y4d still kill the scrub's own save/restore.

Knob-off PRGs byte-identical to master 4e45a9b (sha256, built before/after on this tree: git checkout 4e45a9b then the branch)

profile sha256 (before == after)
ip65 3fccbeb13d343b55d5a9a4fc7a59e9a4a09b9a3aaee26b65c497bc8f34a4354e
ip65-onchip e69bfce7b552e765df02cb253b4ff12cecaca3e978669376b7af5ffad9343dce
uci fe3b6c204d95fcdb3e818fcdc43275b72bd8ec5adc14fb28f7dfe26902846d20
uci-onchip 9e11f570f219e7fcb6bd9ba0f8398ce8fee28f930d8022941c74d6c7f1f1f9e6
uci-comb ad85797b67320b81441d6a651c12c164465e642c26fea346fc4e3d55a13e4a14
uci-onchip-store 1e82434885d87aa4e7a34d57230dc805759ddf18d3a081d95a5c7e85d353e70b
uci-comb-store fee4d5b107b2f848bdaa41b6855aef3f2a2032920cf5a893fe8c71b2fa3e2f2b

(make / USE_NISTCURVES_ONCHIP=1 / BACKEND=uci / +USE_NISTCURVES_ONCHIP=1 / +USE_NISTCURVES_ONCHIP_COMB=1 / +TRUST_STORE=1 variants; make clean; sleep 1 between builds. Equal to the #314 table.)

PQ images (non-SNI), before (4e45a9b) -> after

profile before after
uci-onchip-pq 208f3967d119b4ac3ca85c80435f81cdc6e07ac4b6d14b11b5cb9c3377b4bb93 354f70d0a8cffe1cca4ef32d28e8721e7e753d65bd925a425b023d881be7cd6b
uci-comb-pq e97ab54a730f24e6ecd8988eabe71120649b3cac9200194871c4e6cdb95332b8 b8677dd79968fb57e475baaa0995c8979d670b159ae753b207a947d5e8be8d61
uci-onchip-store-pq eeeb8b0f9e24b4962a0a135a8b243ecefc79c90a2bf65e664fa4ac3dca9e09bc 3b4dd718cbdc296f2083c90ce4d89a25e1fbd032e849c66f25227ef4d0ac4d5a
uci-comb-store-pq 0efe08030abc7d74e3aa449733c9f637ddbcefe6c15480cb29273a6848680252 41202de9fa6b9da769fce87a3223e7d1c74630f873b4b64a940afacddb7cbeb6

Margins (python3 tools/measure_margins.py --build --profile uci-onchip-pq --profile uci-comb-pq --profile uci-onchip-store-pq --profile uci-comb-store-pq, free bytes, #314 -> now)

profile LOADER NET_CODE CRYPTO_OVERLAY CRYPTO_HOT
uci-onchip-pq 179 -> 179 166 -> 105 2,578 509
uci-comb-pq 457 -> 457 335 -> 272 325 505 -> 507
uci-onchip-store-pq 157 -> 157 166 -> 105 372 196
uci-comb-store-pq (tightest) 342 -> 342 197 -> 134 55 377 -> 379

Cost: about 63 B of TLS_CODE (NET_CODE), 2 B back from the pq_sh_decaps cleanup (CRYPTO_HOT on comb). The tightest region on uci-comb-store-pq is CRYPTO_OVERLAY (55 B, unchanged); NET_CODE on the store profiles is down to 105-134 B. Comb loses 63 B of NET_CODE, onchip 61 B.

Red -> green

  • RED on master 4e45a9b's PQ image (git checkout 4e45a9b -- src/tls_keyschedule.s src/crypto/pq_hybrid.s; python3 tools/test_pq_keyschedule.py): RESULTS: 28/48 passed, 20 failed. Every $EC check against the host HKDF fails (handshake secret, both traffic secrets, 4 key/iv buffers, twice), the secrets are not zeroed; the $1D and group-0 schedule checks pass on master (the classic path).
  • GREEN on this branch: python3 tools/test_pq_keyschedule.py: 48/48. The fixture plants tls_mlkem_ss, tls_shared_secret, group and transcript, runs tls_derive_handshake_keys, and compares with hashlib/hmac HKDF over the host-known IKM; the host references for hybrid / classic / swapped-order differ (asserted).
  • tools/test_pq_kem.py [4] (tls_connect end to end) also asserts, at a breakpoint on tls_derive_handshake_keys, that the two secrets and the transcript are the host's, and afterwards tls_handshake_secret == host HKDF over K || ss (11EC) / ss (001D), both secrets zero.
  • Oracle: tools/test_pq_keylog_check.py red cases per input (either secret, swapped client/server, residue in each of seed/ss/shared secret, missing keylog lines, 7 gate arms incl. "CONNECTED then closed reads tls_state=0" green and "ERROR with a stale latch" green).

Tests (all on this head, exact commands)

command result
python3 tools/test_pq_keyschedule.py 87/87 (round 1: adds groups $EB/$ED/$FF)
python3 tools/test_pq_sh.py 190/190
C64_SKIP_BUILD=1 python3 tools/test_pq_kem.py 84/84
python3 tools/test_mlkem_placement.py 194/194
python3 tools/test_mlkem_linked.py (with mutants) 34/34
python3 tools/test_pq_keylog_check.py / python3 tools/pq_keylog_check.py --selftest / python3 tools/test_openssl_listener.py 11/11, SELFTEST PASS, 2/2
BACKEND=uci python3 tools/test_hs_sequence.py 62/62
BACKEND=uci python3 tools/test_tls_handshake.py 36/36
python3 tools/test_keyschedule_steps.py / test_hkdf.py / test_finished_verify.py 9/9, 12/12, 18/18
python3 tools/test_runner_coverage.py / test_pytest_boundary.py PASSED, PASSED
python3 tools/test_build_flags_stamp.py 21/21
python3 -m pytest -q 510 passed

Mutants (python3 tools/mutate_pq_sh.py --no-control K01..K12 M17 M23 M43 M44 M49 Y3p Y4d: 19/19 killed, control after restore green; each dies by an assertion)

id mutation killed by
K01 IKM order x25519 || mlkem EC handshake_secret != host
K02 IKM length 32 not 64 EC handshake_secret
K03 group test inverted EC handshake_secret
K04 compares $1D instead of $EC EC handshake_secret
K05 hybrid never selected EC handshake_secret
K06 IKM pointer still tls_shared_secret EC handshake_secret
K07 concat one byte short EC handshake_secret
K08 x25519 ss not zeroed EC tls_shared_secret zero
K09 mlkem ss not zeroed EC tls_mlkem_ss zero
K10 input_buffer not zeroed EC input_buffer[0:64] zero
K11 hmac_data_buf not zeroed IKM window survives (offsets 55,56)
K12 zeroing only 32 of 64 B EC input_buffer[0:64] zero
X02 (round 1) hybrid for any group >= $EC (bne->bcc) ED handshake_secret != host (classic expected)
M17,M23,M43,M44,M49,Y3p,Y4d existing P4 mutants touching the edited code (M49's pattern updated for the edited comment) as before

mutate_pq_sh.py now reports a non-zero exit without an assertion FAIL line as CRASH-OR-HANG (not a kill). The full 56 P4 mutants were not re-run (only the ones whose code I touched); tools/mutate_pq_sh.py controls now include the new suite.

Hardware (U64E 10.43.23.81 via DeviceLock, 48 MHz, OpenSSL 3.6.5 s_server, free ports 24510-24512, 4433 untouched)

Local listener = tools/https_e2e/openssl_listener.py + tools/uci/rig_https_local.py with EXTERNAL_LISTENER=1 EXTERNAL_PORT=<p> PQ_DUMP=<dir>; then python3 tools/pq_keylog_check.py --state <dir>/pq_state.json --keylog <dir>/keylog.txt. "OpenSSL group" = the ServerHello key_share group in s_server -trace.

Images (both built from d226402's src; later commits are tools/docs):

  • onchip: make BACKEND=uci USE_NISTCURVES_ONCHIP=1 PQ_HYBRID=1 HTTPS_SNI=www.foo.invalid sha256 ab85014cdc8bb074fc9cb99180c90dc8a74ac4d8ed9f68cfda32e7d311cf5e0f
  • comb (REU 16 MB, C64_INIT_WAIT=90): make BACKEND=uci USE_NISTCURVES_ONCHIP_COMB=1 PQ_HYBRID=1 HTTPS_SNI=www.foo.invalid sha256 265c4c583c2b5a2430daf3a7ee2a3e546d99a8b7c1c4173476e8455097cfa042
image server -groups OpenSSL group tls_kex_group result rig wall-clock oracle
onchip X25519MLKEM768 11EC (4588) $EC full HTTPS PASS (HELLO FROM TLS SERVER) 108 s MATCH (exit 0)
onchip X25519 ecdh_x25519 (29) $1D full PASS 107 s inconclusive 78 (not a hybrid run)
onchip default (no -groups) 11EC $EC full PASS 109 s MATCH
comb X25519MLKEM768 11EC $EC full PASS 165 s MATCH
comb X25519 29 $1D full PASS 164 s inconclusive 78
comb default 11EC $EC full PASS 166 s MATCH

With PHASE_TIMING=1 (comb, 1 s poll, n=1 each): trigger to client Finished 43.7 s hybrid vs 42.0 s X25519-only on the same image, i.e. about +1.7 s (~4%) for KeyGen + Decaps + the bigger flight. Rig wall-clock above includes boot/prep and is within 1-2 s between groups.

boot_check.py PASS on both images.

Round 1, store path: make BACKEND=uci USE_NISTCURVES_ONCHIP=1 TRUST_STORE=1 PQ_HYBRID=1 HTTPS_SNI=www.foo.invalid, sha256 46398f0bd258de344f75f2df8994b7bbf8c913d1a59cd7054db5442c4e43d400, local -groups X25519MLKEM768 (port 24513): full HTTPS PASS, OpenSSL group 11EC, rig 107 s, oracle MATCH (cert_pin_hs_keys -> tls_derive_handshake_keys). rig_https_local.py needs no trust files (its trampoline sets tp_mode, tools/uci/_trust_mode.py).

Round 1 no-code-change proof: the 7 knob-off hashes, the 4 non-SNI PQ hashes and both SNI hardware hashes (ab85014c..., 265c4c58...) rebuilt identical after the comment/test changes. Docs now carry the image hashes, make lines and timings (engineering-notes P5 section); CLAUDE.md distinguishes local (onchip+comb) from live (comb only; wikipedia = HTTP 301 at /). OpenSSL 3.6.5 with no -groups picks X25519MLKEM768 for our hello.

Bonus, real public servers (tools/uci/rig_https_live.py, comb image without HTTPS_SNI: make BACKEND=uci USE_NISTCURVES_ONCHIP_COMB=1 PQ_HYBRID=1, sha256 b8677dd79968fb57e475baaa0995c8979d670b159ae753b207a947d5e8be8d61):

  • browserleaks.com: PASS, tls_kex_group=$EC, HTTP 200, tls_reached_connected=7, client Finished at 44.1 s (the non-PQ comb image ad85797b...: 42.3 s).
  • en.wikipedia.org (EXPECT_STATUS=301, /): PASS, $EC, HTTP 301, 39.8 s.
  • github.com, lwn.net: openssl s_client -groups X25519MLKEM768 gets handshake_failure (no 11EC); not run on the C64.
  • Trap found on the way (documented in engineering-notes): a PQ image built with HTTPS_SNI=www.foo.invalid fails a LIVE host with df_last_err=$05 (SAN checked against the baked name). It looked like a hybrid defect for one run; the same image without the override passes.

Known gaps

  • The oracle compares application traffic secrets, so it proves the whole schedule (hs secret -> master -> app secrets through the server Finished) but cannot attribute a mismatch to ML-KEM vs X25519 vs transcript; the host fixture in test_pq_keyschedule.py is what pins the IKM order.
  • n=1 for the timing delta; only the comb image was phase-timed (onchip comparison is rig wall-clock only).
  • A server that sends the 1,178 B SH as ONE record still cannot be read (P4 gap, unchanged).
  • The ip65 backend, uci-m3 and the uci REU profile remain $(error) for PQ_HYBRID.
  • Only the mutants touching changed code were re-run, not all 56 P4 mutants.

🤖 Generated with Claude Code

JC-000 and others added 4 commits October 17, 2026 17:42
…racle on app traffic secrets, tests and mutants

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…and df_last_err

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ILLED requires a FAIL line (no code byte change)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@JC-000
JC-000 marked this pull request as ready for review October 18, 2026 00:16
@JC-000
JC-000 merged commit 4987049 into master Oct 18, 2026
@JC-000
JC-000 deleted the feat/pqc-p5-hybrid-keyschedule branch October 18, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant